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The  Military  Operations  Research  Society 

(MORS) 


The  purpose  of  the  Military  Operations  Research  Society  is  to  enhance  the  quality  and 
effectiveness  of  classified  and  unclassified  military  operations  research.  To  accomplish 
this  purpose,  the  Society  provides  media  for  professional  exchange  and  peer  criticism 
among  students,  theoreticians,  practitioners,  and  users  of  military  operations  research. 
These  media  consist  primarily  of  the  traditional  annual  MORS  symposia  (classified), 
their  published  proceedings  and  abstracts,  special  mini-symposia,  workshops,  colloquia 
special  purpose  monographs  and  other  publications.  The  forum  provided  by  these  media 
is  directed  to  display  the  state  of  the  art,  to  encourage  consistent  professional  quality,  to 
stimulate  communication  and  interaction  between  practitioners  and  users,  and  to  foster 
the  interest  and  development  of  students  of  operations  research.  In  performing  its  func¬ 
tion,  the  Military  Operations  Research  Society  does  not  make  or  advocate  official  policy 
nor  does  it  attempt  to  influence  the  formulation  of  policy.  Matters  discussed  or  state¬ 
ments  made  during  the  course  of  its  symposia  or  printed  in  its  publications  represent  the 
positions  of  the  individual  participants  and  authors  and  not  of  the  Society. 

The  Military  Operations  Research  Society  is  operated  by  a  Board  of  Directors  consisting 
of  30  members,  28  of  whom  are  elected  by  vote  of  the  Board  to  serve  a  term  of  four 
years.  The  persons  nominated  for  this  election  are  normally  individuals  who  have  at¬ 
tained  recognition  and  prominence  in  the  field  of  military  operations  research  and  who 
have  demonstrated  an  active  interest  in  its  programs  and  activities.  The  remaining  two 
members  of  the  Board  of  Directors  are  the  Past  President  who  serves  by  right  and  the 
Executive  Vice  President  who  serves  as  a  consequence  of  his  position.  A  limited  number 
of  Advisory  Directors  are  appointed  from  time  to  time,  usually  a  1  -year  term,  to  perform 
some  particular  function.  Since  a  major  portion  of  the  Society's  affairs  is  connected  with 
classified  services  to  military  sponsors,  the  Society  does  not  have  a  general  membership 
in  the  sense  that  other  professional  societies  have  them.  The  members  of  MORS  are  the 
Directors,  persons  who  have  attended  a  MORS  meeting  within  the  past  three  years  and 
Fellows  of  the  Society  (FS)  who,  in  recognition  of  their  unique  contributions  to  the  Soci¬ 
ety,  are  elected  by  the  Board  of  Directors  for  life. 

MORS  is  sponsored  by: 

•  The  Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

•  The  Director  Assessment  Division,  Office  of  the  Chief  of  Naval  Operations 

•  The  Director  of  Modeling,  Simulation  and  Analysis,  Deputy  Chief  of  Staff,  Plans  and 
Operations,  Headquarters,  US  Air  Force 

•  Commanding  General,  Marine  Corps  Combat  Development  Command 

•  The  Director  of  Force  Structure,  Resource  and  Assessment,  The  Joint  Staff 

•  The  Director  Program  Analysis  and  Evaluation,  Office  Secretary  of  Defense 
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The  Society  of  Cost  Estimating  and  Analysis 

(SCEA) 


The  purpose  of  the  Society  of  Cost  Estimating  and  Analysis  is  to  further  the  effectiveness 
and  efficiency  of  cost  estimating  and  analysis  and  related  disciplines  in  the  public  and 
private  sectors  by: 

•  Promoting  and  enhancing  the  profession  of  cost  estimating  and  analysis. 

•  Fostering  the  professional  growth  of  its  members. 

•  Enhancing  the  understanding  and  appreciation  of  cost  estimating,  analysis  and 
related  disciplines  throughout  the  general  public. 

•  Promoting  a  common  body  of  knowledge  as  a  standard  for  individual  excellence. 

•  Advocating  a  uniform  code  of  ethics  for  the  profession. 

•  Rewarding  achievement  through  an  appropriate  program  of  recognition  and  certi¬ 
fication. 

•  Providing  forums  and  media  through  which  experiences  with  the  principles  and 
techniques  of  cost  estimating  and  analysis  may  be  reported,  discussed  and  pub¬ 
lished  in  furtherance  of  public  interest. 

•  Fostering,  promoting,  and  conducting  research  and  educational  programs. 

•  Developing  and  maintaining  standards  of  proficiency  and  performance. 

•  Cooperating  with  other  organizations  and  individuals,  having  common  or  related 
purposes,  in  furtherance  of  public  interest. 

•  Providing  an  opportunity  for  government,  industry  and  academia  to  collectively 
discuss  and  comment  on  related,  proposed  or  suggested  subjects  of  common  in¬ 
terest. 

•  Establishing  standards  in  the  terminology,  conduct  and  application  of  cost  esti¬ 
mating  and  analysis. 

Membership  is  open  to  all  interested  individuals  in  the  cost  estimating  and  cost  analysis 
professions  and  related  disciplines. 

Members  are  from  all  levels  of  management,  are  experts,  journeymen,  and  beginners 
firom  the  government,  private  sector  and  academia. 
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PREFACE 


This  conference  is  the  first  conference  jointly  sponsored  by  MORS  and  SCEA.  Both  the 
operations  research  and  cost  communities  have  a  mutual  and  overlapping  interest  in  per¬ 
forming  analysis  required  for  COEAs.  We,  as  communities,  know  how  to  do  effective¬ 
ness  analysis  and  we  know  how  to  do  cost  analysis.  However,  linking  or  integrating  the 
two  presents  us  with  a  imique  challenge.  It  seemed  only  natural  that  our  two  communi¬ 
ties  should  join  forces  to  address  the  COEA  costing  topic,  acknowledging  and  recogniz¬ 
ing  that  we  should  work  together  to  solve  our  common  problems.  It  was  apparent,  from 
the  other  workshops  and  symposia  on  COEAs  sponsored  by  OSD/P A&E,  that  integrating 
both  types  of  analysis  was  an  essential  ingredient  for  successful  completion,  review  and 
approval  at  the  highest  levels.  Thus,  the  idea  for  this  conference  was  bom. 
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1.  Introduction 


1.1  BACKGROUND 

During  a  time  of  rapid  changes  in 
the  world  and  constrained  resources,  it  is 
of  the  utmost  importance  that  Depart¬ 
ment  of  Defense  (DoD)  decision  makers 
be  supported  by  pertinent  and  timely 
analysis.  In  an  attempt  to  enhance  and 
upgrade  the  level  of  analysis,  the  Diree- 
tor  of  Program  Analysis  and  Evaluation, 
Office  Secretary  of  Defense 
OSD(PA&E)  developed  and  published 
Cost  and  Operational  Effectiveness 
Analysis  (COE A)  guidelines  as  part  of 
the  then  new  5000  Series  Regulations  to 
provide  a  framework  for  conducting 
COEAs. 

A  series  of  workshops  was  con¬ 
ducted  to  help  explain  the  relationship  of 
the  COEA  to  the  acquisition  decision 
making  process.  The  first  workshop  was 
held  on  3  April  1991  at  the  Defense 
Systems  Management  College,  Fort  Bel- 
voir,  Virginia  and  included  senior  offi¬ 
cials  and  analysts  from  DoD  and  the 
military  departments.  Seventy-two  gen¬ 
eral  offieers,  senior  executive  service 
civilians  and  others  from  all  services  met 
to  discuss  problems  with  current 
COEAs.  Each  workshop  was  opened  by 
Dr.  David  Chu,  ASD(PA&E).  Short 
talks  by  subject  matter  experts  were 
given  with  eaeh  followed  by  a  brief  dis¬ 
cussion  period.  Each  service,  in  turn, 
presented  its  unique  problems  and  per¬ 
ceptions  of  the  workshop. 

Three  "limited  attendance"  action 
officer  workshops  were  held  in  May 
1991,  in  McLean,  Virginia  at  the  MITRE 
Corporation  with  the  MITRE  Economic 


Analysis  Center  serving  as  host.  Each 
workshop  aimed  to  provide  a  forum  for 
working  level  analysts  to  discuss  meth¬ 
ods  for  improving  analyses  associated 
with  COEAs.  Dr.  Chu  provided  opening 
comments  by  stressing  the  need  for  bet¬ 
ter  analyses.  Workshop  topics  were  pre¬ 
sented,  again  by  subjeet  matter  experts. 

A  tutorial  eonceming  the  PA&E 
workshops  was  presented  at  the  1991 
Annual  MORS  Symposium  at  the  US 
Military  Academy  in  West  Point,  New 
York.  The  tutorial  focused  on  the  con¬ 
duct  of  the  workshops,  on  issues  gener¬ 
ated  from  the  workshops  and  on  trends 
or  perceptions  resulting  from  the  work¬ 
shops. 

A  two  and  one-half  day  mini¬ 
symposium  focusing  on  COEAs  in  the 
acquisition  process  was  held  in  March 
1992  in  Newport,  Rhode  Island,  with 
MORS  serving  as  sponsor.  Dr.  Chu  pre¬ 
sented  the  keynote  address,  once  again 
expressing  the  need  and  importance  of 
COEAs.  Congressman  Ron  Machtley  of 
Rhode  Island  provided  the  luncheon 
address  noting,  in  particular,  the  need  to 
develop  an  analytical  basis  for  allocating 
scarce  defense  resources. 

Results  from  all  workshops  were 
reported  at  the  1992  Annual  MORS 
Symposium  at  the  Naval  Postgraduate 
School  in  Monterey,  California.  The 
need  to  focus  on  the  costing  aspects  of 
COEAs  was  highlighted  during  these 
sessions.  In  all  the  workshops,  tutorials 
and  symposiums,  interest  from  all  seg¬ 
ments  of  DoD  and  contractor  personnel 
was  high. 
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1.2  OBJECTIVES 

The  objective  of  the  MORS/ 
SCEA  mini-symposium  was  to  examine 
the  role  and  methodology  of  cost  analy¬ 
sis  in  the  COEA  process  as  well  as  to 
examine  the  application  of  operations 
research  techniques  useful  to  the  inte¬ 
gration  of  both  cost  and  effectiveness 
analysis.  Exploring  the  broader  aspect 
of  affordability  analysis  is  another  one  of 
the  basic  objectives  of  the  mini¬ 
symposium.  In  particular,  the  goals 
were  to: 

•  Understand  the  role  of  COEAs  in 
the  decision  making  process 

•  Examine  the  role  of  cost  analysis 
in  the  COEA  process 

•  Establish  a  set  of  common  cost 
analysis  issues  and  problems 
faced  when  performing  COEAs 

•  Develop  possible  solutions  or 
identify  appropriate  research  ar¬ 
eas  common  to  those  cost  analy¬ 
sis  issues  and  problems 

•  Improve  the  collaborative 
framework  for  dealing  with 
COEA  cost  analysis  require¬ 
ments 

1.3  ORGANIZATION 

The  mini-symposium  covered  a 
two  and  one-half  day  period  and  pro¬ 
vided  a  forum  for  addressing  the  cost 
analysis  issues  and  problems  related  to 
COEAs. 

There  were  four  major  addresses, 
including  a  keynote  speech  by  Dr.  David 
Chu.  There  were  two  panel  discussions 
—  one  a  senior-level  panel  which  ad¬ 
dressed  the  general  issue  of  cost  esti¬ 
mating  as  part  of  the  COEA  process. 


with  emphasis  on  the  policy  perspec¬ 
tives  of  the  services  and  OSD,  and  the 
other  a  mid-level  panel  which  placed 
emphasis  on  the  implementation  per¬ 
spectives  of  the  services.  Finally,  four 
working  groups,  each  addressed  a  spe¬ 
cific  topic  area. 

1.3.1  Principal  Speakers.  The  princi¬ 
pal  speakers  were: 

•  Dr.  David  Chu,  RAND  and  for¬ 
mer  Assistant  Secretary  of  De¬ 
fense  (Program,  Analysis  and 
Evaluation)  was  the  keynote 
speaker. 

•  Dr.  David  McNicol,  Deputy  As¬ 
sistant  Secretary  (Resource 
Analysis),  OASD  (PA&E),  and 
Chairman  of  the  Cost  Analysis 
Improvement  Group; 

•  Mr.  Frank  Kendall,  Chair,  Con¬ 
ventional  Systems  Committee, 
Defense  Acquisition  Board; 

•  Mr.  Robert  Soule,  Deputy  Di¬ 
rector,  Acquisition  Resources, 
Acquisition  Policy  and  Program 
Integration. 

1.3.2  Working  Groups.  Four  working 
groups  were  convened  to  help  explain 
the  relationship  of  the  COEA  to  the  ac¬ 
quisition  decision  making  process.  Short 
talks  by  subject  matter  experts  were 
given  with  each  followed  by  a  brief  dis¬ 
cussion  period.  The  four  working 
groups  were:  Comparative  Cost  Analysis 
and  Methodology,  Uncertainty  Analysis, 
Integrating  Cost  and  Effectiveness  and 
Modeling  Cost  and  Performance. 

1.3.3  Panels.  Two  panels  convened 
following  Dr.  Chu's  remarks.  The  panels 
consisted  of  representatives  from  each  of 
the  services'  cost  organizations  as  well  as 
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OSD(PA&E).  The  panels  addressed  the 
general  issue  of  cost  estimating  as  part 
of  the  COEA  process.  Particular  empha¬ 
ses  were  on  the  policy  perspectives  of 
the  services  and  OSD. 

Senior  Panel.  The  Senior  Panel 
consisted  of  the  senior  person  from  each 
of  the  services'  cost  organizations  as  well 
as  OSD(PA&E).  The  panel  addressed 
the  general  issue  of  cost  estimating  as 
part  of  the  COEA  process.  Particular 
emphases  were  on  the  policy  perspec¬ 
tives  of  the  services  and  OSD. 

Each  of  the  panel  members  pro¬ 
vided  a  senior-level  perspective  on  the 
cost  issues  resulting  from  institutional¬ 
izing  COEAs. 

Each  of  the  service  representa¬ 
tives  raised  similar  concerns  and  echoed 
the  others  comments.  They  identified 
workload  and  personnel  issues  as  one  of 
their  primary  concerns  with  performing 
"quality  and  credible"  COEAs.  They 
also  identified  the  Cost  Analysis  Re¬ 
quirements  Document  (CARD)  as  a  use¬ 
ful  document  for  facilitating  COEA 
costing,  provided  it  was  completed  in  a 
timely  manner. 

From  the  OSD  level,  one  of  the 
primary  issues  was  ensuring  the  services 
included  sufficient  alternatives  to  sub¬ 
stantiate  the  COEA  results.  That  is, 
"types  of  system  alternatives"  as  well  as 
"alternatives  to  meeting  the  requirement" 
other  than  a  weapon  system  acquisition. 

Participants  in  the  panel  were: 

•  Moderator  —  Dr  Stephen  Balut, 

Director,  Economic  Analysis 

Center,  Institute  For  Defense 


Analyses. 

•  OSD  —  Dr  David  McNicol, 
Deputy  Director  (Resource 
Analysis)  OSD(PA&E)  and 
Chairman  of  the  Cost  Analysis 
Improvement  Group  (CAIG). 

•  Army  —  Mr.  Robert  Young,  Di¬ 
rector,  US  Army  Cost  and  Eco¬ 
nomic  Analysis  Center. 

•  Navy  —  Captain  Richard  S. 
Coleman,  Director,  Naval  Center 
for  Cost  Analysis. 

•  Air  Force  —  Mr.  Lee  Baseman, 
Deputy  Assistant  Secretary  of  the 
Air  Force  (Economic  Analysis) 

Mid-level  panel.  The  panel  ad¬ 
dressed  issues  of  cost  estimating  as  part 
of  the  COEA  process.  Particular  empha¬ 
ses  were  on  the  implementation  perspec¬ 
tives  of  the  services.  Each  of  the  panel 
members  provided  a  senior-level  per¬ 
spective  on  the  cost  issues  resulting  from 
institutionalizing  COEAs. 

Participants  in  the  panel  were: 

•  Moderator  —  Maj  Sylvia 
Wardley-Niemi,  Office  of  the 
Assistant  Secretary  of  the  Air 
Force  (Financial  Management). 

•  Navy  —  Ms.  Noreen  Bryant,  Di¬ 
rector  of  Cost  Analysis,  Naval 
Air  Systems  Command. 

•  Army  —  Dr.  Herb  Fallin,  Di¬ 
rector,  Assessment  and  Evalua¬ 
tion,  Office  of  the  Assistant  Sec¬ 
retary  of  the  Army  (RDA). 

•  Air  Force  —  Mr.  Jack  Graser, 
Office  of  the  Assistant  Secretary 
of  the  Air  Force  (Financial  Man¬ 
agement). 

•  OSD  —  Mr.  Lance  Roark,  Office 

of  Deputy  Director  (Resource 
Analysis),  OSD(PA&E). 
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2.  Working  Group  I 

Comparative  Cost  Analysis  and  Methodology 

Dr  Daniel  A.  Nussbaum,  Naval  Center  for  Cost  Analysis,  and  Mr.  Leonard  S.  Freeman, 
Office  Chief  of  Naval  Operations  (OP-81) 


2.1  GOALS  AND  ISSUES 

Cost  estimating  is  important  and 
adds  value  to  the  acquisition  process.  In 
fact,  the  cost  estimating  process  en¬ 
hances  understanding  of  the  program  by 
forcing  greater  clarity  in  program  defini¬ 
tion.  Early  involvement  by  the  cost  es¬ 
timating  community  is  critical  for  credi¬ 
ble  and  useful  COEAs.  At  the  previous 
MORS  mini-symposium  on  COEAs, 
everyone  agreed  that  special  emphasis 
should  be  placed  on  the  fact  that  the  first 
letter  in  COEA  is  a  "C",  and  it  stands  for 
Cost.  Without  proper  and  early  attention 
to  cost,  the  COEA  process  is  fatally 
flawed.  The  purpose  of  the  Comparative 
Cost  Analysis  Working  Group  was  to 
identify  the  commonly  perceived  prob¬ 
lems,  as  well  as  the  thoughts  of  the 
costing  part  of  the  COEA  world. 

This  working  group  addressed 
the  following  issues: 

•  The  validity  of  cost  estimating 
relationships  (CERs)  —  There  is 
a  constant  need  to  update  data 
bases  in  order  to  reflect  current 
technology  and  acquisition  envi¬ 
ronments. 

•  Updating  cost  tools  —  This  is  a 
continuous  process  requiring 
people  and  funds.  Cost  estimat¬ 
ing  tools  are  perishable  as  the  ac¬ 
quisition  environment  changes 
(e.g.,  business  base,  acquisition 
strategy,  etc.) 


2.2  SUMMARY 

There  were  10  presenters  in  the 
working  group.  They  represented  all  the 
services  in  DoD  and  the  FAA.  Each 
speaker  had  actually  done  the  cost 
analysis  for  the  COEA  they  were 
presenting.  The  speakers  provided  a 
short  overview  of  the  COEA  —  the 
options,  who  did  the  work,  and  the 
results  —  and  then  addressed  the  set  of 
common  questions. 

A  common  set  of  questions  was 
discussed  by  each  presenter,  to  act  as  a 
focusing  agent  on  the  working  group 
issues. 

Following  the  presentations,  the 
working  group  analyzed  the  central 
tendency  of  the  papers.  These  coalesced 
into  a  set  of  four  issues: 

•  The  independence  of  the  cost 
estimate 

•  The  need  for  a  cost  validation 
authority 

•  The  need  for  an  oversight  board 

•  A  gauge  of  the  appropriate  effort 
to  be  put  into  a  COEA. 

We  take  these  four  issues  up,  in 
turn,  below. 

The  independence  of  the  cost 
estimate  —  DoDI  5000.2  provides 
guidance  on  doing  COEAs,  and  it 
requires  that  the  cost  estimates  in  the 
COEA  be  "consistent"  with  the  cost 
estimates  presented  at  the  program's 
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milestone  review.  The  working  group 
participants  believe  that  this  consistency 
requirement  makes  it  inevitable  that  the 
cost  estimates  that  the  COEA  analysts 
use  will  come  from  the  PM,  and  that 
they  will  be  indistinguishable  from  the 
most  recent  POM/budget  inputs  from  the 
program  office.  The  concern  of  the 
working  group  participants  is  that 
consistency  has  replaced  validity  as  the 
driving  criterion. 

The  need  for  a  cost  validation 
authority  —  Concern  about  the  validity 
of  the  cost  estimate  led  to  a  recognition 
that  there  has  to  be  a  validating  authority 
analogous  to  the  independent  cost 
estimate  (ICE)  required  at  a  milestone 
review.  Is  there  a  validating  mechanism 
in  the  Services?  What  we  found  was 
that: 

•  The  Army  produces  an  Army 
Cost  Position  (ACP),  which  is 
the  result  of  a  review  of  the  costs 
in  the  COEA  and  any  other  cost 
estimates  for  the  program.  By 
the  ACP,  the  Army  does  validate 
the  COEA  cost  estimate. 

•  The  Air  Force  validates  its 
COEA  cost  estimates  by  filtering 
the  estimates  through  the  Air 
Force  Cost  Analysis  Improve¬ 
ment  Group  (AFCAIG).  In  this 
way,  the  Air  Force  formally 
validates  the  COEA  cost 
estimate. 

•  The  Navy  has  no  formal  valid¬ 
ating  process.  It  is  true  that  the 
Naval  Center  for  Cost  Analysis 
(NCA)  serves  as  the  Navy's 
independent  cost  estimator  for 
milestone  purposes,  and  sits  on 
the  Navy  COEA  oversight 
Boards  (in  an  advisory  capacity 


only),  and  from  that  perch  can 
advise  on  proper  cost  estimating 
standards.  Nevertheless,  there's 
no  formal  process  in  the  Navy  to 
validate  the  cost  estimate  in  the 
COEA. 

The  need  for  an  Oversight  Board 
(OB')  —  Each  Service  has  a  group  whose 
function  is  to  oversee  the  accomplish¬ 
ment  of  the  COEA.  The  Army  calls  it  a 
Study  Advisory  Group  (SAG),  the  Air 
Force  calls  it  a  COEA  Advisory  Board 
(CAG),  and  the  Navy  calls  it  a  COEA 
Oversight  Board  (COB).  Whatever  it  is 
called,  almost  every  speaker  in  the 
working  group  avowed  the  importance 
of  the  Oversight  Group,  and  made  the 
following  points: 

•  Early  involvement  of  the  OB  is 
important  to  the  success  of  the 
COEA. 

•  Active  participation  of  the  OB  is 
very  important. 

•  A  Study  Plan,  briefed  to  and  ap¬ 
proved  by  the  OB,  is  indis¬ 
pensable  to  the  success  of  the 
COEA.  Two  of  the  sections  of 
the  Study  Plan  that  were  emph¬ 
asized  by  the  working  group 
speakers  were: 

o  Resource  section,  identifying 
the  resources  (time,  dollars, 
schedule,...)  needed  for  the 
COEA. 

o  Cost  section,  identifying  the 
relevant  cost  issues  to  be 
addressed  in  the  COEA. 

A  gauge  of  the  anpropriate  effort 
for  a  COEA  —  A  number  of  speakers 
emphasized  the  importance  of  making 
sure  that  the  effort  put  into  the  COEA  is 
commensurate  with  the  problem  at  hand. 
In  particular,  the  number  of  options. 
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paying  attention  to  the  milestone 
decision,  and  relative  versus  absolute 
costs  are  thought  to  be  important  issues. 
COEAs  should  be  tailored  to  the 
circumstances. 
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3.  Working  Group  II 
Uncertainty  Analysis 

Dr  Richard  Trainor,  US  Army  Cost  and  Economic  Analysis  Center 


3.1  Goals  and  Issues 

Trade-offs  between  alternatives 
offering  varying  levels  of  effectiveness 
vis  a  vis  different  costs  are  specifically 
addressed  by  the  COEA.  The  choice 
between  competing  alternatives  is  often 
aided  by  using  uncertainty  analysis. 
Uncertainty  in  the  context  of  the  sympo¬ 
sium  refers  to  the  major  factors  that  can 
be  expected  to  adversely  impact  the  ac¬ 
curacy  of  future  cost  estimates  and 
thereby  undercut  the  credibility  of  the 
resulting  COEAs. 

The  working  group  attempted  to 
identify  these  factors,  describe  their  im¬ 
pact  on  cost  analysis  accuracy,  identify 
actions  required  to  reduce  future  cost 
analysis  uncertainty  and  address  ways  of 
applying  uncertainty  analysis  to  COEA 
cost  analysis. 

3.2  Summary 

Due  to  the  rapidly  changing  na¬ 
ture  of  the  world,  it  was  decided  to  treat 
the  subject  of  uncertainty  analysis  in  a 
broadly  based  geo-political  context 
rather  than  in  its  more  conventional 
mathematical  orientation.  Therefore  this 
working  group  focused  mainly  on  how 
cost  analyses  10-15  years  in  the  future 
will  be  affected  by  changing  world 
events.  The  changes  will  occur  gradu¬ 
ally.  However,  their  cumulative  effect  is 
nearly  certain  to  be  major  over  the  long 
term. 


Seven  papers  were  presented  in 
the  Uncertainty  Analysis  Working 
Group.  They  represented  the  views  of 
the  Army,  Navy,  NASA,  and  both  profit 
and  non-profit  analysis  organizations. 
The  content  of  the  papers  covered  wide 
ranging  topics  as  was  appropriate  to  this 
study  group. 

Of  the  many  problem  areas  that 
can  be  expected  to  cause  future  cost  un¬ 
certainty,  five  were  highlighted  in  this 
working  group.  These  problem  areas  are 
the  data  base,  the  industrial  base,  acqui¬ 
sition  policies,  availability  of  CERs  and 
the  COEA  workload. 

The  availability  of  a  robust,  ap¬ 
plicable  data  base  is  fundamental  to 
sound  cost  analyses.  However,  as  we 
produce  fewer  weapon  systems  and  in 
smaller  quantities  the  data  base  shrinks 
accordingly.  More  effort  is  required,  to 
include  more  inter  service  cooperation, 
to  assure  that  maximum  use  is  being 
made  of  the  shrinking  data  base. 

As  the  military  budgets  are  re¬ 
duced  in  size,  we  can  expect  to  see  an 
increasing  need  to  assess  the  impact  of 
non-recurring  costs  and  semi-variable 
costs.  These  cost  categories  can  be  ex¬ 
pected  to  attain  increasing  importance 
since  companies  in  aerospace  industry 
base  can  be  expected  to  have  difficulties 
in  directing  their  talents  to  the  develop¬ 
ment  and  production  of  "plowshares." 
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Emerging  acquisition  policies 
such  as  "silver  bullets"  and  developing 
weapon  systems  and  then  "putting  the 
technology  on  the  shelf  may  have  merit. 
However,  the  emerging  acquisition  poli¬ 
cies  have  cost  implications  and  not  all  of 
these  are  obvious  at  first  glance.  The 
cost  analysis  community  should  take  the 
lead  to  ascertain  the  full  cost  conse¬ 
quences  of  key  emerging  acquisition 
policies. 

The  final  major  problem  dis¬ 
cussed  in  this  work  group  was  the 
COEA  workload.  As  the  Department  of 
Defense  shrinks  in  size  we  can  expect 
fewer  COEAs.  However,  this  reduction 
will  occur  very  slowly,  probably  not  as 
fast  as  reductions  in  the  size  of  cost 
analysis  staffs.  Therefore,  the  surviving 
cost  analysts  can  expect  an  increasing 
workload.  Moreover,  future  COEAs  will 
focus  increasingly  on  analyses  that  occur 
during  the  Milestone  0  to  Milestone  1 
period.  During  this  period  the  alterna¬ 
tives  are  poorly  defined.  There  is  no 
Cost  Analysis  Requirements  Document 
(CARD)  and  top  management  wants  the 
results  yesterday.  This  dilemma,  if  it  is 
to  be  solved,  will  require  an  entirely  new 
cost  analysis  methodology.  This  new 
methodology  hopefully  will  result  in  top 
level  CERs  that  will  allow  the  cost 
analysis  community  to  participate  in 
these  important,  but  fast  moving  deci¬ 
sion  processes.  A  major  research  effort 
will  be  required  to  determine  whether 
such  top  level  CERs  can  be  developed. 
Such  research  should  be  given  a  high 
priority  within  OSD  and  by  each  military 
department. 
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4.  Working  Group  III 
Integrating  Cost  and  Effectiveness 

Mr.  Wilbur  C.  Hogan  and  Ms.  Mary  Henry,  US  Army  Training  and  Doctrine  Command 

(TRADOC) 


4.1  Goals  and  Issues 

The  purpose  of  this  working 
group  was  to  examine  various  methods 
of  integrating  cost  and  effectiveness  and 
to  discuss  their  respective  merits  and 
faults.  This  working  group  was  also  to 
address  how  the  Cost  Analysis  and  Op¬ 
erations  Research  professions  are  suc¬ 
cessfully  integrating  cost  and  effective¬ 
ness  analysis  such  that  decision  makers 
can  understand  and  use  the  analysis  to 
make  decisions. 

DoD  5000.2M  states  "there  is  no 
magic  formula  for  combining  cost  and 
effectiveness  measures  to  identify  a  pre¬ 
ferred  alternative."  Accordingly,  the 
services  have  been  striving  to  come  up 
with  acceptable  methods. 

This  working  group  examined 
those  methods  and  discussed  their  merits 
and  faults.  A  series  of  papers  was  pre¬ 
sented  as  food  for  thought  on  different 
approaches  to  accomplishing  this  critical 
task.  One  series  of  two  papers  looked  at 
Value-Added  Analysis  as  a  means  of 
comparing  two  or  more  COEAs  dealing 
with  different  means  of  overcoming  the 
same  deficiency  or  meeting  the  same 
requirement.  The  co-chairs  presented  a 
paper  dealing  with  accepted  methods 
currently  used  in  COEAs. 

4.2  Summary 


this  working  group  including  represen¬ 
tatives  from  the  Defense  Systems  Man¬ 
agement  College,  The  MITRE  Corpora¬ 
tion,  RAND,  and  various  Army  analysis 
organizations. 

There  were  two  central  issues 
identified  throughout  the  sessions.  First, 
methods  for  integrating  cost  and  effec¬ 
tiveness  are  still  evolving,  and  second, 
there  is  "no  single  best  method"  for  inte¬ 
gration. 


There  were  nine  presentations  in 
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5.  Working  Group  IV 
Modeling  Cost  And  Performance 

Dr.  John  G.  Honig,  Management  Analysis  Incorporated  (MAI) 


5.1  Goals  and  Issues 

Cost  performance  analysis  (as 
differentiated  from  cost  effectiveness 
analysis)  is  a  critical  factor  in  design 
trade-off  analysis  of  weapon  systems  and 
support  equipment.  A  relationship  of 
performance  parameters  to  cost  and  ef¬ 
fectiveness  needs  to  be  understood  and 
can  frequently  be  modeled. 

This  working  group  addressed 
the  modeling  of  performance  parameters 
and  their  relationship  to  cost  and  effec¬ 
tiveness.  It  was  arranged  into  five  topi¬ 
cal  sessions,  which  dealt  with  an  Intro¬ 
duction  and  definition  of  context,  Mac¬ 
roeconomics,  Case  studies,  Design-to- 
Cost,  and  a  General  discussion  session 
on  the  whole  working  group  topic.  A 
summary  report  on  the  working  group 
was  delivered  in  a  final  general  session. 
In  addition,  at  least  one-half  hour  was  set 
aside  in  each  session  to  discuss  the  par¬ 
ticular  topic,  and  the  time  was  generally 
used  constructively  with  many  working 
group  attendees  participating. 

5.2  Summary 

Not  all  "COEAs"  are  truly 
"COEAs."  The  early  COEAs  identified 
by  Mr.  Kendall  are  required  to  select  the 
best  approaches.  If  more  than  one  Serv¬ 
ice  can  perform  the  mission  who  is  the 
"honest  broker"  that  defines  missions, 
measures  of  effectiveness  correctly  and 
reviews  that  study  results  evenly?  Rough 
system  definitions  will  produce  rough 


operational  effectiveness  estimates  and 
rough  cost  estimates. 

Requirements  documents  specify 
performance  parameters  that  should  be 
met  for  a  system  to  achieve  a  given  op¬ 
erational  effectiveness.  As  the  opera¬ 
tional  assumptions  which  are  the  basis 
for  the  operational  effectiveness  change, 
who  is  responsible  for  analyzing  the  im¬ 
pact  of  the  system's  inherent  perform¬ 
ance  on  the  new  operational  effective¬ 
ness? 

As  technological  changes  and 
cost  constraints  impact  the  performance 
parameters  of  a  system  under  develop¬ 
ment,  who  analyzes  the  impact  of  those 
performance  changes  on  operational 
effectiveness? 

At  Milestone  0  the  approaches 
are  so  poorly  defined  that  performance 
measures  are  difficult  to  specify.  How 
few  performance  parameters  is  the  ana¬ 
lyst  willing  to  accept  to  produce  a  rough 
estimate? 

Decision  makers  would  like  to 
have  macro  parametric  models  that  esti¬ 
mate  costs  related  to  performance  pa¬ 
rameters.  Given  the  instability  of  CERs 
and  the  absence  of  good,  homogenous 
data  bases  is  such  a  process  feasible? 

The  importance  of  cost  analyst 
involvement  as  early  in  the  process  as 
possible  was  emphasized.  The  cost 
analyst  needs  to  be  part  of  the  altema- 
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lives  definition  process  to  assure  that 
enough  parameters  are  defined  to  be  able 
to  cost  the  system. 

Design-to-cost  will  be  increas¬ 
ingly  important  as  budgets  get  tighter 
and  each  development  competes  with 
many  others  for  scarce  resources.  It  is 
important  that  the  impact  of  performance 
changes,  driven  by  budget  constraint,  on 
operational  effectiveness  be  evaluated 
rapidly.  That  is,  it  is  important  to  deter¬ 
mine  whether  an  aircraft  with  speed  or 
maneuverability  degradation  of  15%  is 
still  operationally  effective  compared  to 
the  aircraft  it  replaces. 

The  cost  analyst  needs  to  work 
closely  with  the  design  engineers  to  pro¬ 
vide  responsive  feedback  on  the  impact 
of  design  changes  on  cost.  Once  a  de¬ 
sign  is  complete,  it  is  sometimes  difficult 
to  incorporate  changes  required  to  lower 
the  system's  cost. 

Measures  of  Effectiveness  are 
parameters  of  the  degree  of  accomplish¬ 
ing  specified  missions.  These  are  opera¬ 
tional  measures  of  achieving  the  mission 
objective.  These  measures  are  a  function 
of  operational  parameters  to  include  a 
selection  of  missions  to  be  accom¬ 
plished,  operational  seenarios,  enemy 
threat  characteristics  and  tactics,  strate¬ 
gies  and  tactics  involved  in  the  use  of  the 
system  being  analyzed  and  others.  Op¬ 
erational  scenarios  include  consideration 
of  offensive  or  defensive  action,  daylight 
or  night  operations  and  others  that  im¬ 
pact  the  success  of  accomplishing  a  mis¬ 
sion. 

Measures  of  Performance  are 
parameters  that  are  intrinsic  to  a  system, 
based  on  its  design  and  manufacture. 


These  characteristics  are  generally 
measurable,  e.g.,  air  speed,  altitude, 
cross  country  speed,  target  detection 
range,  hit  probability,  etc.  These  meas¬ 
ures  are  relatively  independent  of  the 
operational  scenario  in  which  they  are 
employed.  These  measures  are  also  the 
ones  used  by  engineers  to  design  a  sys¬ 
tem  and,  consequently  are  used  by  cost 
analysts  to  cost  a  system.  Cost  analysts 
do  not  cost  operational  effectiveness, 
they  cost  performance. 

As  operational  contexts  change, 
for  example,  with  the  demise  of  the  So¬ 
viet  threat,  operational  effectiveness  is 
said  to  ehange.  In  reality,  "effectiveness" 
does  not  change,  but  utility  of  the  system 
changes  as  the  threat  disappears. 
"Operational  effectiveness"  can,  there¬ 
fore,  be  considered  a  utility  measure. 

As  the  threat  changes  the  inherent 
characteristics  of  systems,  their  perform¬ 
ance  does  not  change,  and  eonsequently 
their  acquisition  cost  does  not  change.  If 
the  systems  utility  is  decreased  pre¬ 
sumably  the  systems  operational  tempo 
(usage)  will  decrease,  and  in  a  life  cycle 
cost  sense,  the  operating  and  support 
costs  will  decrease. 

A  key  issue,  that  was  not  dis¬ 
cussed  in  detail,  but  is  an  overriding 
factor  in  this  discussion  is  the  relation¬ 
ship  between  effectiveness  and  perform¬ 
ance.  It  was  stated  above  that  the  opera¬ 
tional  effectiveness  changes  with  opera¬ 
tional  parameters.  It  is  important  to  se¬ 
lect  representative  operational  parame¬ 
ters  that  lead  to  a  determination  of  the 
desired  performance  parameters.  In  the 
test  and  evaluation  process  the  perform¬ 
ance  parameters  are  generally  measured, 
rather  than  the  operational  effectiveness. 
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although  the  effectiveness  is  considered 
qualitatively  during  the  evaluation  in  an 
operational  sense. 

There  is  a  corollary  to  this  issue. 
As  budget  constraints  increase  severely, 
cost  and  performance  tradeoffs  must  be 
analyzed.  For  example,  how  much  sav¬ 
ings  can  be  expected  if  a  less  capable 
tank  engine  is  used,  decreasing  cross¬ 
country  speed  by  15%.  Someone  needs 
to  analyze  the  impact  of  this  lesser  speed 
on  the  operational  effectiveness  of  the 
future  tank. 

A  statement  found  in  DoDD 
5000.2,  Part  4,  Section  E,  Paragraph 
3.a.(5)  is: 

"To  judge  whether  an  alternative  is 
worthwhile,  one  must  first  determine 
what  it  takes  to  make  a  difference. 
Measures  of  effectiveness  should  be  de¬ 
fined  to  measure  operational  capabilities 
in  terms  of  engagement  or  battle  out¬ 
comes.  Measures  of  performance,  such 
as  weight  and  speed,  should  relate  to  the 
measure  of  effectiveness  such  that  the 
effect  of  a  change  in  the  measure  of  per¬ 
formance  can  be  related  to  a  change  in 
the  measure  of  effectiveness." 

In  order  to  get  insight  on  how 
cost  and  effectiveness  are  integrated  Mr. 
Hogan,  TRADOC,  provided  a  talk  on  the 
subject.  It  was  the  same  talk  he  also  gave 
in  his  working  group.  His  bottom  line 
was  that  there  is  no  good  way  of  inte¬ 
grating  cost  and  effectiveness.  He  dis¬ 
cussed  in  some  detail  problems  with 
determining  effectiveness.  He  arrived  at 
the  concept  of  sufficiency,  i.e.,  what  is 
the  system  being  analyzed  supposed  to 
accomplish,  but  he  found  no  consistent 
methods  for  determining  sufficiency. 


The  further  question  still  remains  of  how 
to  translate  sufficiency  into  performance 
parameters. 

Mr.  Dennis,  SPARTA,  discussed 
the  influence  of  scenarios  on  perform¬ 
ance.  He  illustrated  his  talk  with  exam¬ 
ples  from  the  Strategic  Defense  Initiative 
context.  He  discussed  the  impact  of  at¬ 
tack  parameters  on  such  measures  as 
timeliness,  engagement  leverage  and 
cost  per  intercept.  He  concluded,  based 
on  his  analyses,  that  "scenario  selection 
can  drive  costs  to  meet  requirements,  by 
unequal  focus  on  performance  attrib¬ 
utes." 

Macroeconomics.  The  first  talk 
was  by  ETC  Loerch,  US  Army  Concepts 
Analysis  Agency,  who  presented  his 
concept  for  a  Value  Added  Analysis 
(VAA).  He  defined: 

"Value  Added:  The  incremental  return 
on  investment  as  measured  using  explicit 
effectiveness  values  and  implicit  effec¬ 
tiveness  values  as  compared  to  cost. 

“Explicit  Measures  of  Effectiveness: 
Objective  factors  that  measure  the  worth 
of  the  system/program  in  terms  of  its 
contribution  to  overall  force  effective¬ 
ness  (e.g.,  combat  simulation  results). 

“Implicit  Measures  of  Effectiveness: 
Subjective  factors  (e.g.,  political  risk) 
that  affect  the  decision  making  process." 

This  concept  was  clearly  and 
cogently  presented,  and  is  certainly  use¬ 
ful  in  the  overall  decision  making  proc¬ 
ess.  However,  since  this  working  group 
was  primarily  concerned  with  perform¬ 
ance  rather  than  effectiveness,  and  it  was 
already  stated  that  cost  really  relates  to 


15 


performance,  the  question  remains  how 
well  differences  in  performance  of  indi¬ 
vidual  weapon  systems,  e.g..  reduction 
in  cross-country  speed  impact  the  results 
of  VAA.  Differences  in  performance  can 
result  from  technological  improvements, 
manufacturing  difficulties  and  cost  con¬ 
straints,  for  example.  How  sensitive  is 
the  VAA  methodology  to  analyzing  al¬ 
ternatives  with  different  performance 
characteristics. 

Another  comment  should  be 
made  regarding  the  omission  of  operat¬ 
ing  and  support  costs  in  the  optimization 
process.  It  was  indicated  that  this  esti¬ 
mate  was  difficult  to  arrive  at  because  of 
uncertainties  in  estimating  operating  and 
support  costs  of  developmental  systems. 
On  the  other  hand,  it  is  well  known  that 
operating  and  support  costs  represent  a 
major  portion  of  the  life  cycle  cost. 
Therefore  estimates  of  operating  and 
support  costs  are  provided  to  the  deci¬ 
sion  maker  as  information,  but  these 
costs  are  not  included  in  the  optimization 
model. 

Mr.  Daigle,  US  Army  Tank  and 
Automotive  Command,  discussed 
Analysis  of  Residual  Value,  Military 
Usefulness,  Economic  and  Military  Use¬ 
ful  Life.  Mr.  Daigle  is  in  the  Tactical 
Wheeled  Vehicle  Fleet  Planning  Office 
and  is  concerned  with  the  replacement  of 
truck  and  other  support  vehicles.  He  has 
developed  mathematical  algorithms  to 
determine  the  useful  life  of  a  vehicle 
based  on  cumulative  maintenance  costs 
and  the  investment  of  replacing  the  ag¬ 
ing  vehicle  with  a  new  vehicle  of  the 
same  kind.  He  determined  residual  value 
and  economic  or  militarily  useful  life 
based  on  operating  and  support  costs  and 
age  characteristics  of  the  vehicle  to  be 


replaced. 

It  would  appear  that  the  applica¬ 
bility  of  Mr.  Daigle's  analyses  are 
somewhat  unique  to  his  office.  In  his 
commodity  are  a  number  of  critical 
things  that  do  not  change  significantly. 
The  missions  of  a  truck  in  support  of 
Army  units,  and  the  scenario  in  which 
they  operate  are  rather  fixed.  Techno¬ 
logical  advances  are  generally  not  radi¬ 
cal.  The  replacement  vehicle  will  proba¬ 
bly  not  differ  significantly  from  the  ag¬ 
ing  vehicle  being  replaced.  It  probably 
would  be  useful  to  analyze  the  feasibility 
of  extending  the  basic  concept  to  a  wider 
variety  of  commodity  classes. 

Case  Studies.  Mr.  Denelsbeck, 
Frontier  Technology,  Inc.,  lead  a  discus¬ 
sion  on  "Cost  Architecture  for  Advanced 
Design  (CAFAD)."  CAFAD  is  an  engi¬ 
neering-based  cost  architecture  used  for 
design-to-cost,  tradeoffs  among  design 
alternatives,  pre-Milestone  0  level  cost¬ 
ing.  Its  objective  is  to  arrive  at  the  best 
performance  within  a  cost  constraint.  It 
is  an  automated  system,  uses  an  interdis¬ 
ciplinary  approach,  and  relies  on  a  close 
working  relationship  between  the  design 
engineers  and  cost  analysts.  It  permits 
direct  feedback  when  trading  off  per¬ 
formance  goals  and  can,  therefore,  di¬ 
rectly  link  enhanced  or  reduced  perform¬ 
ance  to  cost. 

In  the  pre-Milestone  0  environ¬ 
ment  the  designer  works  largely  off  the 
MENS  which  is  frequently  very  general 
and  includes  performance  parameters 
which  are  frequently  not  based  on  hard 
analyses.  The  relative  accuracy  of  the 
cost  estimates  depends  on  the  accuracy 
of  the  design  parameters  based  on  the 
MENS. 
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Mr.  Denelsbeck  discussed  in 
some  detail  the  approach  that  was  taken 
to  develop  this  cost  architecture.  It  is 
based  on  an  overall  system  executive 
routine,  made  up  of  concept,  in¬ 
put/output  and  cost  executive  routines. 
Further  details  of  each  of  these  routines 
as  well  as  the  input  data  requirements 
were  discussed. 

In  the  final  analysis,  the  model 
should  be  capable  to  link  engineering 
buildups  to  specific  engineering  pa¬ 
rameters  that  drive  costs.  The  model  will 
therefore  provide  costs  for  advanced 
concepts  from  level  0  to  level  3.  Cost 
drivers  will  be  identified  early  in  the 
process,  data  will  be  provided  to  support 
investment  strategy  decisions,  full 
weapon  system  design  and  technology 
are  inserted  at  the  subsystem  level,  and 
the  lab-wide  investment  planning  proc¬ 
ess  will  be  supported. 

Mr.  Harmon,  Cost  Analysis  and 
Research  Division,  Institute  for  Defense 
Analyses,  examined  the  relationship 
between  performance  and  cost  for  vari¬ 
ous  methods  of  integrating  avionics  in 
the  next  generation  of  tactical  aircraft. 
The  two  principal  methods  that  were 
compared  was  the  traditional  federated 
architecture  of  combining  building 
blocks  consisting  of  various  electronic 
functions  on  the  one  hand,  and  a  fully 
integrated  set  of  avionics  that  incorpo¬ 
rates  the  same  functions  in  a  single  sys¬ 
tem  on  the  other  hand. 

In  the  integration  process  de¬ 
creased  hardware  life  cycle  costs  are 
traded  off  against  increased  software  life 
cycle  costs  resulting  from  increases  in 
functionality.  Integration  will  result  in 
lower  airframe  and  engine  life  cycle 


costs  and  potential  increases  in  weapon 
system  effectiveness  resulting  from 
software  improvements.  On  the  cost 
side,  the  integrated  system  costs  are  due 
more  to  increased  software  capacity  and 
complexity.  It  is  expected  that  develop¬ 
ment  costs,  integration  costs  and  support 
costs  will  all  increase. 

Hardware  life  cycle  cost  savings 
were  estimated  based  on  cost  estimating 
relationships  developed  for  processor 
weights  over  time  (historical  data),  and 
estimating  the  weight  of  federated  sys¬ 
tem  architecture  and  the  integrated  sys¬ 
tem,  as  well  as  the  impact  of  those 
weight  differences  on  aircraft  weight  and 
thrust.  Hardware  life  cycle  cost  savings 
could  then  be  estimated. 

Cost  estimating  relationships  and 
other  relationships  were  used  to  estimate 
the  development  cost,  the  integration 
cost  and  operating  and  support  cost  for 
integrated  software.  Costs  were  derived 
for  physieal  integration  (traditional  fed¬ 
erated  architecture)  and  three  levels  of 
functional  integration. 

Design-To  Cost.  Design-to-cost 
was  defined  in  DoDD  5000.28(encl.  2) 
as: 

"Design  to  Cost  fPTCh  An  acqui¬ 
sition  management  technique  to  achieve 
system  designs  that  meet  stated  require¬ 
ments  for  life  cycle  cost  (LCC)  elements. 
Cost  is  a  key  system  eonsideration  that  is 
addressed  on  a  continuing  basis  as  an 
inherent  part  of  the  development  and 
production  process.  The  technique  em¬ 
bodies  early  establishment  of  realistic 
but  rigorous  cost  objectives,  goals,  and 
thresholds  and  a  determined  effort  to 
achieve  them". 
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Furthermore,  the  same  document 
states  that: 

"The  achievement  of  DTC  parameters 
shall  be  considered  as  important  as 
achievement  of  performance,  schedule, 
manpower,  and  supportability  require¬ 
ments." 

The  above  document  has  been 
superseded  by  a  number  of  later  publi¬ 
cations,  and  DTC  was  finally  incorpo¬ 
rated  into  DoDD  5000.2.  Nowhere  have 
the  above  quotations  been  restated  as 
clearly  in  later  documents.  However, 
DoDD  5000.2,  Part  6,  Section  K,  states: 

"These  policies  and  procedures  establish 
cost  as  a  design  constraint  early  in  the 
acquisition  life  cycle." 

With  increasingly  tighter  budget 
constraints,  the  need  to  develop  future 
systems  within  tight  cost  constraints  will 
become  very  important.  This  requires 
not  only  the  capability  to  trade  off  cost 
and  performance,  but  again  requires  un¬ 
derstanding  of  how  performance  degra¬ 
dation  impacts  operational  effectiveness. 

Mr.  Covert,  TECOLOTE,  dis¬ 
cussed  a  Survivability  Cost  Estimating 
Model  (SCEM),  and  then  proceeded  to 
demonstrate  the  computer  model  to  the 
working  group.  SCEM  was  developed 
using  Cost  Engineering  Integrated  Tools 
(CEIT).  A  key  factor  in  making  this 
model  work  is  a  close  integration  of  the 
design  and  cost  analysis  process.  The 
latter  is  considered  an  integral  part  of  the 
systems  engineering  process.  Rather  than 
being  handed  a  system  to  cost,  after 
having  been  completely  designed,  the 
cost  analyst  works  with  the  engineer 
every  step  of  the  way. 


In  the  application  shown,  SCEM 
was  applied  to  Javelin.  Hardening  tech¬ 
niques  to  counter  nuclear  and  laser 
threats  were  examined.  Each  subsystem 
had  a  suite  of  hardening  options  that 
could  be  selected,  and  each  selection  had 
performance,  weight  and  cost  impact.  By 
modeling  these  factors  explicitly  the 
impact  of  design  changes  could  be  read¬ 
ily  determined.  By  combining  system 
design  logic  with  cost  estimating  tech¬ 
nologies  the  decision  maker  could  make 
cost-conscious  decisions  concerning 
each  hardening  technique  on  an  element- 
by-element  basis. 

General  Discussion.  The  last 
session  provided  an  opportunity  for 
spirited  discussion  among  many  of  the 
working  group  attendees.  It  became  evi¬ 
dent  that  not  all  COEAs  are  "true 
COEAs".  Mr.  Kendall,  in  his  luncheon 
speech  pointed  out  the  need  for  more 
"COEA"  prior  to  milestone  0.  This 
brought  out  a  number  of  issues. 

If  the  mission  has  the  potential  of 
being  performed  by  more  than  one 
Service  who  is  responsible  for  a  Super- 
COEA,  who  assures  that  each  of  the 
Service  candidates  are  analyzed  on  a 
comparable  basis,  performs  an  inde¬ 
pendent  risk  analysis,  and  measures  that 
the  costs  are  comparable? 

Milestone  0  analyses  lead  to 
specification  of  operational  effectiveness 
and  performance  parameters.  In  most 
cases  there  may  only  be  very  few  of 
these  performance  parameters  and  they 
are  based  on  little  analysis.  Even  the 
effectiveness  analyses  are  macro,  yet  the 
expectations  of  the  cost  estimate  are  of  a 
valid,  tight  estimate.  A  key  issue  facing 
the  cost  analyst  is,  on  the  basis  of  how 
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few  performance  parameters,  will  he  be 
willing  to  put  a  cost  estimate  on  paper. 
He  must  also  consider  that  costs  have  a 
life  of  their  own,  and  whatever  assump¬ 
tions  he  makes  may  be  forgotten  when 
the  costs  are  quoted  in  the  Washington 
Post. 

A  related  subject  deals  with  rela¬ 
tive  cost  estimates.  Early  in  the  design 
cycle  it  may  only  be  possible  to  have 
rough  estimates  of  the  relative  cost  of 
various  approaches.  This  may  be  ade¬ 
quate  to  chose  an  alternative,  but  it  is  not 
adequate  to  determine  its  affordability.  It 
may  be  that  none  of  the  alternatives  are 
affordable  with  serious  redesign  within 
DTC  constraints. 

If  technological  or  cost  risks  find 
an  alternative  to  be  too  risky  and  has  to 
be  redesigned,  who  will  reassess  the 
operational  effectiveness  of  the  degraded 
performance  system?  Who  determines 
that,  given  this  degradation,  how  the 
rank  ordering  of  the  alternatives 
changes? 

The  1992  COEA  conference 
identified  the  following  methodological 
issue: 

"Better  Macro  Level  Parametric  Cost 
Models  are  Needed." 

The  same  item  continues: 

"Parametric  Macro  Cost  Models  —  Per¬ 
formance  Parameter  Based  —  are 
needed  to  allow  cost  trades." 

The  answer  proposed  was  to 
"develop  quick  response  methodologies, 
new  macro  level  models  with  perform¬ 
ance  parameters  available  by  acquisition 


phase,  and  by  commodity  class." 

There  was  no  more  discussion 
about  the  desirability  of  such  models 
than  the  desirability  of  motherhood. 
However,  there  was  little  consensus 
about  even  the  feasibility  of  such  models 
with  a  reasonably  acceptable  degree  of 
uncertainty. 

It  is  evident  that  development  of 
such  macro  models  requires  cost  esti¬ 
mating  relationships  that  continue  to  be 
valid  over  time  and  that  are  based  on 
reasonably  homogeneous  data  bases. 
There  was  consensus  that  neither  of 
these  are  in  existence,  nor  likely  to  be 
happening.  With  rapidly  advancing  tech¬ 
nologies  and  irregular  trend  lines,  ex¬ 
trapolating  CERs  based  on  historical 
data  is  speculative  at  best. 

Greg  Staley,  Air  Force 
ASC/XRE,  discussed  his  organization's 
cost  engineering  process,  whereby  de¬ 
sign  and  cost  estimates  are  developed 
simultaneously.  He  discussed  the  ad¬ 
vantages  of  a  joint  Air  Force-Industry 
Life  Cycle  Cost  Engineering  Working 
Group.  He  also  emphasized  the  impor¬ 
tance  of  performance  impact  on  operat¬ 
ing  and  support  costs.  He  also  empha¬ 
sized  the  importance  of  XR  working 
with  the  using  command  to  refine  the 
requirement  to  have  designable/costable 
parameters. 
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Appendix  1 
Agenda 


2  March  1993 

0730-0830  Registration  (Coffee,  Pastries,  Juices) 

0830-0845  Welcome  /  Introduction  of  Keynote  Speaker 
0845-0930  KEYNOTE  ADDRESS  —  Dr.  David  Chu,  RAND  Corp;  formerly 
Assistant  Secretary  of  Defense,  Program  Analysis  and  Evaluation 
0930-1000  BREAK 

1000-1200  SENIOR-LEVEL  PANEL  DISCUSSION  —  "OSD  CAIG  and  Mili¬ 
tary  Departments  Response  to  COEA  Costing",  moderated  by  Dr. 
Steven  Balut,  Institute  for  Defense  Analysis 
1200-1330  LUNCHEON  with  SPEAKER  (Dr.  McNicol) 

1330-1530  GENERAL  SESSION:  MID-LEVEL  PANEL  DISCUSSION  — 
"Issues  in  COEA  Cost  Analysis” 

1530-1600  BREAK 

1600-1700  GENERAL  SESSION:  AFFORDABILITY  ANALYSIS  —  Robert 
Soule,  Office  of  the  Assistant  Secretary  of  Defense,  Acquisition. 

3  March  1993 

0800-1000  CONCURRENT  WORKING  GROUPS 

Comparative  Cost  Analysis  and  Methodology  (Dr.  Nussbaum  and  Mr. 
Freeman) 

Uncertainty  Analysis  (Dr.  Trainor) 

Integrating  Cost  and  Effectiveness  (Mr.  Hogan  and  Ms.  Henry) 
Modeling  Cost  and  Performance  (Dr.  Honig) 

1000-1030  BREAK 

1030-1200  CONCURRENT  WORKING  GROUPS 

1200-1330  LUNCHEON  With  SPEAKER  (Mr.  Frank  Kendall  III) 

1330-1500  CONCURRENT  WORKING  GROUPS 
1500-1530  BREAK 

1530-1700  CONCURRENT  WORKING  GROUPS 

4  March  1993 

0800-1000  CONCURRENT  WORKING  GROUPS 
1000-1030  BREAK 

1030-1200  CONCURRENT  WORKING  GROUPS 
1200-1300  LUNCH 

1300-1500  WORKING  GROUP  REPORTS 
1500-1530  CLOSING  REMARKS 
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Appendix  2 

TERMS  OF  REFERENCE 


BACKGROUND 

During  a  time  of  rapid  changes  in  the  world  and  constrained  resources,  it  is  of  the 
utmost  importance  that  Department  of  Defense  (DoD)  decision  makers  are  supported  by 
pertinent  and  timely  analysis.  In  an  attempt  to  enhance  and  upgrade  the  level  of  analysis, 
OSD/P A&E  developed  and  published  COEA  guidelines  as  part  of  the  new  5000  Series 
Regulations  to  provide  a  framework  for  conducting  COEAs. 

A  series  of  workshops  was  conducted  to  help  explain  the  relationship  of  the 
COEA  to  the  acquisition  decision  making  process.  The  first  workshop  was  held  on  3 
April  1991  at  the  Defense  Systems  Management  College,  Fort  Belvoir,  Virginia  and 
included  senior  officials  and  analysts  from  DoD  and  the  military  departments.  Seventy- 
two  general  officers,  senior  executive  service  civilians  and  others  from  all  services  met  to 
discuss  problems  with  current  COEAs.  The  workshops  were  opened  by  Dr.  David  Chu, 
ASD(PA&E).  Short  talks  by  subject  matter  experts  were  given  with  each  followed  by  a 
brief  discussion  period.  Each  service,  in  turn,  presented  its  unique  problems,  and 
perceptions  of  the  workshop. 

Three  "limited  attendance"  action  officer  workshops  were  held  in  May  1991,  in 
McLean,  Virginia  with  the  MITRE  Corporation  serving  as  host.  Each  workshop  aimed  to 
provide  a  forum  for  working  level  analysts  to  discuss  methods  for  improving  analyses 
associated  with  COEAs.  Dr.  Chu  provided  opening  comments  by  stressing  the  need  for 
better  analyses.  Workshop  topics  were  presented,  again  by  subject  matter  experts. 

A  tutorial  concerning  the  PA&E  workshops  was  presented  at  the  1991  Annual 
MORS  Symposium  at  the  US  Military  Academy  in  West  Point,  New  York.  The  tutorial 
focused  on  the  conduct  of  the  workshops,  on  issues  generated  from  the  workshops  and  on 
trends  or  perceptions  resulting  from  the  workshops. 

A  two  and  one-half  day  mini-symposium  focusing  on  COEAs  in  the  acquisition 
process  was  held  in  March  1992  in  Newport,  Rhode  Island,  with  MORS  serving  as 
sponsor.  Dr.  Chu  presented  the  keynote  address,  once  again  expressing  the  need  and 
importance  of  COEAs.  Congressman  Ron  Machtley  of  Rhode  Island  provided  the 
luncheon  address  noting,  in  particular,  the  need  to  develop  an  analytical  basis  for 
allocating  scarce  defense  resources. 

Results  from  all  workshops  were  reported  to  the  1992  Annual  MORS  Symposium 
at  the  Naval  Postgraduate  School  in  Monterey,  California.  The  need  to  focus  on  the 
costing  aspects  of  COEAs  was  highlighted  during  these  sessions.  In  all  the  workshops. 
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tutorials  and  symposia,  interest  from  all  segments  of  DoD  and  contractor  personnel  has 
been  high. 

OBJECTIVE 

The  objective  of  the  mini-symposium  will  be  to  examine  the  role  and 
methodology  of  cost  analysis  in  the  COEA  process  as  well  as  the  application  of 
operations  research  techniques  useful  to  the  integration  of  both  cost  and  effectiveness 
analysis.  Exploring  the  broader  aspect  of  affordability  analysis  is  another  one  of  the  basic 
objectives  of  the  mini-symposium.  In  particular,  the  goals  are  to: 

•  Understand  the  role  of  COEAs  in  the  decision  making  process; 

•  Examine  the  role  of  cost  analysis  in  the  COEA  process; 

•  Establish  a  set  of  common  cost  analysis  issues  and  problems  faced  when 
performing  COEAs; 

•  Develop  possible  solutions  or  identify  appropriate  research  areas  common 
to  those  cost  analysis  issues  and  problems; 

•  Improve  the  collaborative  framework  for  dealing  with  COEA  cost  analysis 
requirements. 

SCOPE 

The  mini-symposium  will  cover  a  two  and  one-half  day  period  and  will  provide  a 
forum  for  addressing  the  cost  analysis  issues  and  problems  related  to  COEAs.  Dr.  David 
Chu  will  be  the  keynote  speaker.  In  addition,  there  will  be  a  high-level  government  panel 
comprised  of  the  Chairman  of  the  OSD  Cost  Analysis  Improvement  Group  (CAIG)  and 
the  senior  cost  analysis  officials  from  each  of  the  three  services.  The  panel  will  address 
"OSD  CAIG  and  Military  Department  response  to  COEA  Cost  Analysis."  Each  official 
will  give  a  fifteen  to  twenty  minute  presentation,  followed  by  a  fifteen  to  twenty  minute 
period  of  questions. 

Working  groups  on  special  topic  areas  will  be  conducted  in  which  presentations 
will  be  given  by  practitioners  within  the  operations  research  and  cost  analysis 
communities.  The  working  groups  will  be  held  on  the  morning  of  the  second  day  of  the 
symposium  and  repeated  during  the  afternoon  of  the  same  day  to  give  attendees  an 
opportunity  to  participate  in  more  than  one  working  group  session. 

The  mini-symposium  chair  will  select  co-chairs  from  each  services'  cost  centers  as 
well  as  co-chairs  of  the  working  groups  for  the  special  topic  areas.  The  co-chairs  will  in 
turn,  solicit  speakers  to  give  presentations  on  each  of  the  special  topic  areas  in  working 
groups.  The  speakers  will  themselves  be  experts  within  their  respective  areas  and  will 


24 


present  areas  to  explore  and  discuss  the  COEA  issues  facing  cost  analysis  practitioners. 
The  special  topic  areas  to  be  explored  are: 

•  Issues  in  COEA  Cost  Analysis 

•  Methodology  for  Comparative  Cost  Analysis 

•  Uncertainty  Analysis 

•  Integrating  Cost  and  Effectiveness 

•  Modeling  Cost  and  Performance 
Issues  in  COEA  Cost  Analysis 


This  general  session  will  serve,  to  some  extent,  as  an  overview  to  the  more 
detailed  working  group  sessions  which  will  take  place  on  the  second  day  of  the  mini¬ 
symposium.  Hence,  this  special  topic  area  will  be  addressed  before  the  entire  mini¬ 
symposium  audience.  Speakers  from  OSD  and  each  of  the  three  services  will  explain 
which  government  organizations  are  doing  cost  analysis  for  COEAs  and  will  discuss  their 
approaches,  issues  and  problems. 

Methodology  for  Comparative  Cost  Analysis 

Cost  estimating  is  important  and  adds  value  to  the  acquisition  process.  In  fact,  the 
cost  estimating  process  enhances  understanding  of  the  program  by  forcing  greater  clarity 
in  program  definition.  Early  involvement  by  the  cost  estimating  community  is  critical  for 
credible  and  useful  COEAs. 

At  the  last  MORS  mini-symposium  on  COEAs,  everyone  agreed  that  special 
emphasis  should  be  placed  on  the  fact  that  the  first  letter  in  COEA  is  a  "C",  and  it  stands 
for  Cost.  Without  proper  and  early  attention  to  cost,  the  COEA  process  is  fatally  flawed. 

This  working  group  will  address  the  following  issues: 

•  The  validity  of  cost  estimating  relationships  (CERs)  —  There  is  a  constant 
need  to  update  data  bases  in  order  to  reflect  current  technology  and  acquisition 
environments. 

•  Updating  cost  tools  —  This  is  a  continuous  process  requiring  people  and 
funds.  Cost  estimating  tools  are  perishable  as  the  acquisition  environment 
changes  (e.g.,  business  base,  acquisition  strategy,  etc.) 


25 


Uncertainty  Analysis 


Trade-offs  between  altematiyes  offering  yarying  leyels  of  effectiyeness  yis  a  yis 
different  costs  are  specifically  addressed  by  the  COEA.  The  choice  between  competing 
altematiyes  is  often  aided  by  using  uncertainty  analysis.  Uncertainty  in  the  context  of 
the  symposium  refers  to  the  major  factors  that  can  be  expected  to  adyersely  impact  the 
accuracy  of  future  cost  estimates  and  thereby  undercut  the  credibility  of  the  resulting 
COEAs. 

This  working  group  will  attempt  to  identify  these  factors,  describe  their  impact  on 
cost  analysis  accuracy,  identify  actions  required  to  reduce  future  cost  analysis  uncertainty 
and  address  ways  of  applying  uncertainty  analysis  to  COEA  cost  analysis.  This  working 
groups  will  inyolye  four  speakers. 

Integrating  Cost  and  Effectiyeness 

DoD  5000. 2M  states  "there  is  no  magic  formula  for  combining  cost  and 
effectiyeness  measures  to  identify  a  preferred  altematiye."  Accordingly,  the  seryices 
haye  been  striying  to  come  up  with  acceptable  methods. 

This  working  group  will  examine  those  methods  and  discuss  their  merits  and 
faults.  A  series  of  papers  will  be  presented  as  food  for  thought  on  different  approaches  to 
accomplishing  this  critical  task.  One  series  of  two  papers  will  look  at  Value-Added 
Analysis  as  a  means  of  comparing  two  or  more  COEAs  dealing  with  different  means  of 
oyercoming  the  same  deficiency  or  meeting  the  same  requirement.  The  co-chairs  will 
present  a  paper  dealing  with  accepted  methods  currently  used  in  COEAs. 

Modeling  Cost  and  Performance 

Description  still  to  come. 

PARTICIPATION 

Attendance  will  be  limited  to  a  maximum  of  300  people.  The  goal  is  to  get  people 
currently  inyolyed  in  costing,  analyzing  and  modeling  actiyities  associated  with  COEAs 
and  those  responsible  for  proyiding  oyersight  of  the  COEA  and  its  analytical 
methodologies.  The  symposium  will  be  at  the  unclassified  leyel. 

DATE  AND  LOCATION 

The  mini-symposium  will  be  held  March  2-4,  1992  at  the  Fairyiew  Park  Marriott 
Hotel,  Falls  Church,  Virginia. 
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LUNCHEON  SPEAKER 


A  luncheon  will  be  held  at  the  Fairview  Park  Marriott  Hotel  on  each  day  of  the 
symposium.  A  high-level  Government  official  will  be  invited  to  address  the  attendees  on 
the  theme  of  the  symposium  or  other  pertinent  topic  relative  to  the  symposium  theme. 

PRICE 

The  registration  fee  for  the  mini-symposium  will  be  $150.00  for  Government 
personnel  and  $300.00  for  non-government  personnel. 

PRODUCT 

The  product  from  this  mini-symposium  will  include  the  abstracts  from  the  papers 
presented  during  the  special-topic  working  groups  prepared  by  the  presenter.  It  will  also 
include  a  summary  of  each  working  group  and  general  session  done  by  the  session  chair. 
Each  presenter  will  prepare  the  abstract  to  be  included  in  the  published  proceedings 
jointly  published  by  MORS  and  SCEA. 

ORGANIZING  COMMITTEE 

General  Chair 

Mr.  Donald  E.  Mixon 

The  MITRE  Corporation 

(703)  883-6599  FAX:  (703)  883-5963 

Deputy  Chairs 
Dr.  A1  Diaz 

Office  of  the  Secretary  of  Defense,  Program  Analysis  and  Evaluation 
(703)  697-9141  FAX:  (703)  693-5707 

Dr.  Gerald  McNichols 
Management  Consulting  and  Research 
(703)  820-4600  FAX:  (703)  820-4398 

Co-Chairs  (Working  Group  ChairsI 
Issues  in  COEA  Cost  Analysis 

Lance  Roark,  OSD  (PA&E) 

(703)  693-7827;  FAX  (703)  693-5707 

MAJ  Sylvia  Wardley-Niemi 

Air  Force  Cost  Center 

(703)  697-2331;  FAX  (703)  614-9873 
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Methodology  for  Comparative  Cost  Analysis 
Dr.  Daniel  Nussbaum,  Naval  Center  for  Cost  Analysis 
(703)  746-2327;  FAX  (703)  746-2390 

Uncertainty  Analysis 

Dr.  Richeird  J.  Trainor,  US  Army  Cost  and  Economic  Analysis  Center 
(703)  756-1861;  FAX  (703)  756-8732 

Modeling  Cost  and  Performance 
Chair  TBD 

Integrating  Cost  and  Effectiveness 
Wilbur  C.  Hogan,  III  and  Mary  Henry 
US  Army  Training  and  Doctrine  Command 
(804)  728-5803;  FAX  (804)  727-4394 

Other  Members 
Mr.  Leroy  T.  Baseman 
SCEA,  Executive  Director 

Chairman,  Air  Force  Cost  Analysis  Improvement  Group  (AFCAIG) 

Ms.  Natalie.  S.  Addison 

MORS,  Associate  Executive  Director 

Dr.  Tom  Gulledge 
George  Mason  University 

Dr.  David  Lee 

OSD,  Program  Analysis  and  Evaluation 

Ms.  Joan  Lovelace 
MITRE  Corporation 

Major  Verna  McBride 
HQ  AFSAA 

Ms.  Debbie  J.  Vogel 
SCEA  National  Office 

Mr.  Richard  1.  Wiles 
MORS,  Executive  Director 
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Appendix  2 
Participants'^ 

Harold  E  Alston 

US  Army  Intelligence  Threat  Analysis  Ctr 
ATTN;  AIAIT-RTM 
Bldg213,  Stop  314 

Washington  Navy  Yard  DC  20374-5085 
OFF  TEL;  (202)-479-1936  DSN;  335-2625 
FAX;  (202)-488-8846 

DR  Michael  R  Anderberg 

OSD  PA&E 

Rm  2D278  Pentagon 

Washington  DC  20301-1800 

OFF  TEL;  (703)-697-0317  DSN;  227-0317 

FAX;  (703)-693-5707 


Brad  Adams 

Analytical  Services,  Inc  (ANSER) 

1215  Jefferson  Davis  Highway 
Suite  800 

Arlington  VA  22202 
OFF  TEL;  (703)-685-3289 
FAX;  (703)-685-3225 

Doug  Adams 

Center  for  Naval  Analyses 
4401  Ford  Ave 
P.O.  Box  16268 
Alexandria  VA  22302-0268 
OFF  TEL;  (703)-824-2255 
FAX;  (703)-824-2949 

James  L  Adams 
ASCA'XF 
Bldg  2041 
25 1 1  L  Street 

Wright-Patterson  AFB  OH  45433-7503 
OFF  TEL;  (513)-255-4216  DSN;  785-4216 
FAX;  (513)-785-838I 
E-mail;  adamsjl 

Mark  J  Adams 
TRW  One  Space  Park 
Bldg  01,  Room  1271 
Redondo  Beach  CA  90278 
OFF  TEL;  (310)-814-6858 
FAX;  (3 10>8 13-1642 

Scott  M  Allard 

Management  Consulting  &  Research,  Inc. 
Suite  300 

1505  Farm  Credit  Drive 
McLean  VA  22102 
OFF  TEL;  (703)-506-4600 
FAX;  (703)-820-4398 


Denise  M  Anderson 

Naval  Air  Systems  Command 

Washington  DC  20361-5260 

OFF  TEL;  (703)-692-l  198  DSN;  222-3443 

FAX;  (703)-746-2965 

John  C  Anderson 
Consultant 

OFF  TEL;  (703)-971-7144 

Kathleen  A  Anderson 
VEDA,  Inc 

5 1 00  Springfield  Pike,  Suite  320 
Dayton  OH  45431-1289 
OFF  TEL;  (513)-476-3250 
FAX;  (513)-256-1673 

MAJ  Timothy  R  Ascani 

US  Army  Cost  &  Economic  Analysis  Center 

5611  Columbia  Pike 

Falls  Church  VA  22041-5050 

OFF  TEL;  (703)-756-0349  DSN;  289-0349 

FAX;  (703)-756-7552 


Addresses  as  of  April  6, 1996. 
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Eileen  L  Ashby 

Marine  Corps  Rsch  Development  &  Acq  Comd 
Code  C2A 

2033  Barnett  Ave,  Suite  315 

Quantico  VA  22134-5080 

OFF  TEL:  (703)-640-2645  DSN:  278-2645 

FAX:  (703)-640-2655 

Larry  L  Austin 
SAIC 

FOB  46565 

Washington  DC  20050 

OFF  TEL:  (703)-693-0410  DSN:  223-0410 

FAX:  (703)-693-1161 

E-mail:  postlla@aol.com 

Mark  H  Awtry 
TASC 

55  Walkers  Brook  Drive 
Reading  MA  01867 
OFF  TEL:  (6 1 7)-642-2000 
FAX:(617)-944-3138 

CDR  Dennis  R.  Baer 

Naval  Center  for  Cost  Analysis 

NCA-54 

1 1 1 1  Jefferson  Davis  Hwy,  Suite  400 

Arlington  VA  22202-4306 

OFF  TEL:  (703)-604-0307  DSN:  664-0307 

FAX:  (703)-604-0315 

E-mail:  dbaer@dmso.dtic.dla.mil 

William  Baker 

Naval  Air  Systems  Command 

142 1  Jefferson  Davis  Highway 

Washington  DC  2036 1 

OFF  TEL:  (703)-692-9182  DSN:  222-9182 

FAX:  (703)-692-9600 

Capt  John  Barth 

Harry  J  Bartosik 
Vanguard  Research  Inc 
Suite  450 
10306  Eaton  Place 
Fairfax  VA  22030-2201 
OFF  TEL:  (703)-934-6300 
FAX:  (703)-273-9398 

Leroy  T  Baseman  III 
SAF/ACC 

Pentagon,  Room  4D159 
Washington  DC  20330-5018 
OFF  TEL:  (202)-693-0792 


Heidi  W  Baskin 
HQ  ACC/FMAMB 
216  Sweeney  Blvd 
Suite  212 

Langley  AFB  VA  23665-2792 

OFF  TEL:  (804)-764-5431  DSN:  574-5431 

Larry  J  Beasley 
HQ  ASC/XPE 

Wright-Patterson  AFB  OH  45433-6503 
OFF  TEL:  (513)-255-4377  DSN:  785-4377 
FAX:  (513)-255-0650 

Harold  E  Bechthold 

HQ  AFSPACECOM/FMAL 

150  Vandenberg  Street,  #1105 

Peterson  AFB  CO  80914-4010 

OFF  TEL:  (719)-554-5661  DSN:  692-5661 

FAX:  (719)-554-3648 

Peter  M  Beck 
Decision  Technology 
Suite  903 

1300  Crystal  Drive 
Arlington  VA  22209 
OFF  TEL:  (703)-4 16-2523 
FAX:  (703)-416-2523 
E-mail:  pmbeck@ix.netcom.com 

Emmet  R  Beeker 
GRC  International  Inc. 

1 900  Gallows  Road 
Vienna  VA  22182 

OFF  TEL:  (703)-695-0350  DSN:  225-0350 
E-mail:  ebeeker@grci.com 

James  R  Behne 

TRADOC  Analysis  Command-Lee 

Attn:  ATRC-LS 

401  First  Street 

Fort  Lee  VA  23801-6000 

OFF  TEL:  (804)-765-1838  DSN:  539-1838 

FAX:  (804)-539-1456 

E-mail:  behnej@trac.army.mil 

Mary  T  Benze 

Office  of  Aerospace  Studies 

COEA  Support  Division 

3550  Aberdeen  SE 

Kirtland  AFB  NM87117 

OFF  TEL:  (505)-846-8302  DSN:  246-8302 

FAX:  (505)-846-4668 
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Robert  J  Bielgaski 

Naval  Surface  Warfare  Center 

WO  Det 

Silver  Spring  MD  20903 

OFF  TEL:  (30 1)-3 94-21 10  DSN:  290-2110 

FAX:  (301)-394-4722 

Steven  M  Biemer 

Johns  Hopkins  University/APL 

Applied  Physics  Laboratory 

Johns  Hopkins  Road 

Laurel  MD  20723 

OFF  TEL:  (301)-953-8605 

FAX:  (301)-953-5910 

Robert  L  Black 

Gary  R.  Bliss 
OSD  (PA&E) 

Pentagon  Room  2C310 

Washington  DC  20301-1800 

OFF  TEL:  (703)-695-4348  DSN:  227-2999 

FAX:  (703)-693-5707 

Mark  S  Blouin 

MAJ  Roger  D  Bohnke 
HQ  ACC/FMA 
216  Sweeney  Blvd 
Ste212 

Langley  AFB  VA  23665-2792 

OFF  TEL:  (804)-764-5431  DSN:  574-5431 

FAX:  (804)-764-7560 

Raymond  D  Borkowski 
Naval  Air  Systems  Command 
Washington  DC  20361-5245 
OFF  TEL:  (703)-692-7688 
Ext:  2616 

FAX:  (703)-692-9600 

Ronald  S  Bowen 
Tecolote  Research  Inc 
54  Middlesex  Turnpike 
Bedford  MA  01730-1475 
OFF  TEL:  (617)-275-3014 
FAX:  (617)-275-3407 


CDR  Benjamin  F  Breaux 

Naval  Center  for  Cost  Analysis 

1 1 1 1  Jefferson  Davis  Hwy 

Suite  400  West  Tower 

Arlington  VA  22202-4306 

OFF  TEL:  (703)-604-0289  DSN:  664-0289 

FAX:  (703)-604-0289 

E-mail: 

breaux.benjamin_at_nca@hq.secnau.navy.m 
Steven  W  Brennan 

Space  and  Naval  Warfare  Systems  Com 
Architecture  &  Engineering  Dir  (3 1 1-5) 

245 1  Crystal  Drive 

Arlington  VA  22245-5200 

OFF  TEL:  (703)-602-1724  DSN:  332-1724 

FAX:  (703)-602-5891 

E-mail:  brennans@smtp-gw.spawar.navy.mil 
Terry  A  Bresnick 
Fritz  H  Brinck 

Naval  Surface  Warfare  Center 
Dahlgren  Division 
Code A51 

Dahlgren  VA  22448-5100 
OFF  TEL:  (703)-663-7369 
FAX:  (703)-663-7898 

William  A  Brinkley 

Teledyne  Brown  Engineering 

PO  Box  070007 

300  Sparkman  Dr,  MS  170 

Huntsville  AL  35807-7007 

OFF  TEL:  (205)-726-5857 

FAX:  (205)-726-2241 

E-mail:  tony.brinkley@pobox.tbe.com 

CPT  John  D  Bryant 

US  Army  Total  Cost  &  Econ  Analysis  Ctr 
5611  Columbia  Pike 
Room  420 

Falls  Church  VA  22041-5050 

OFF  TEL:  (703)-756-0335  DSN:  284-0335 

FAX:  (703)-756-2625 
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Larry  Buchsbaum 
Naval  Air  Warfare  Center 
A/C  Division,  Code  4.10D  MS86 
PO  Box  5152 

Warminster  PA  18974-0591 

OFF  TEL:  (215)-441-1534  DSN:  441-1534 

FAX:  (215)-441-3932 

E-mail:  buksbaum@nadc.navy.mil 

William  C  Burnham 

US  Army  Logistics  Management  College 

Decision  Sciences  Dept 

Fort  Lee  VA  23801-6050 

OFF  TEL:  (804)-765-4733  DSN:  539-4733 

FAX:  (804)-765-4648 

Marvin  E  Cahill 

The  Analytic  Sciences  Corp  (TASC) 

2555  University  Blvd 
Fairborn  OH  45324 
OFF  TEL:  (513)-426-1040 
FAX:  (513)-426-8888 

Roger  H  Caldow 

Johns  Hopkins  University /A PL 

Johns  Hopkins  Road 

Laurel  MD  20723 

OFF  TEL:  (301)-953-5039 

FAX:  (301)-953-9450 

Joseph  C  Calpin 
The  MITRE  Corporation 
Suite  401,  Crystal  Square  4 
1745  Jefferson  Davis  Highway 
Arlington  VA  22202 
OFF  TEL:  (703)-412-8915 

Jackson  G  Calvert 

US  Army  Space  &  Strategic  Defense  Com 
Attn:  CSSD- 
PO  Box  1500-PI-C 
Huntsville  AL  35802 

OFF  TEL:  (205)-955-4923  DSN:  695-4923 
FAX:  (205)-955-4522 

Daniel  L  Carbo 

Naval  Air  Warfare  Center 

Aircraft  Div- Warminster,  Code  3033 

Street  and  Jacksonville  Rds 

Warminster  PA  18974-5000 

OFF  TEL:  (2 15)-44 1-7006  DSN:  441-7006 

FAX:  (215)-441-1967 

E-mail:  dancarbo@nadc.nadc.navy.mil 


Joseph  P  Cardarelli 
NAVAIRSYSCOM 
AIR-524 

Washington  DC  20361 
OFF  TEL:  (703)-692-7688 
Ext:  2625 

FAX:  (703)-692-9600 


Amanda  JA  Cardiel 
NAVSURFWARCENDIV 
17320  Dahlgren  Rd 
A-51 

Dahlgren  VA  22448-5100 

OFF  TEL:  (703)-663-7369  DSN:  249-7369 

FAX;  (703)-663-7898 

Mary  JoAnn  Carroll 

AFSAA/SAM 

1570  Air  Force  Pentagon 

Washington  DC  20330-1570 

OFF  TEL:  (703)-697-1622  DSN:  227-6902 

FAX:  (703)-697-3441 

E-mail:  carroll@afsaa.hq.af.mil 

Jim  Cha 

Brian  Chappel 
Tecolote  Research,  Inc 
3601  Aviation  Blvd 
Suite  1600 

Manhattan  Beach  CA  90266 
OFF  TEL:  (310)-536-001 1 
FAX:  (310)-536-9922 

E.  Christophe  Chart ier 
Naval  Air  Systems  Command 
Washington  DC  20361-5260 
OFF  TEL:  (703)-692-3447 
FAX:  (703)-746-2965 

DR  David  S  C  Chu 
RAND 

2100  M  Street,  NW 
Washington  DC  20037 
OFF  TEL:  (202)-296-5000 
FAX:  (202)-296-7960 
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Mark  W  Clark 

Martin  Marietta  Air  Traffic  Systems 
475  School  Street,  SW 
MS-V62 

Washington  DC  20024 
OFF  TEL:  (202)-646-5826 
FAX:  (202)-646-2255 

Richard  G  Cline 
Rockwell  International 
12214  Lakewood  Blvd 
Mail  Code:  SX05 
Downey  CA  9024 1 
OFF  TEL:  (310)-797-1060 
FAX:  (310)-797-1516 

Capt  Richard  L  Coleman  (Ret.) 

Craig  E  College 
OSD  (PA&E) 

The  Pentagon,  Room  2D3 1 1 
Washington  DC  20301-1800 
OFF  TEL:  (703)-697-2936 

LCDR  Justin  J  Comstock 
Secretary  of  the  Air  Force 
Space  Systems 
The  Pentagon  4C 1 052 
Washington  DC  20370 
OFF  TEL:  (703)-6 14-0404 

Timothy  C.  Coons 
Sverdrup  Technologies,  Inc 
4200  Col.  Glenn  Highway,  Suite  500 
Dayton  OH  4543 1 

OFF  TEL:  (513)-476-4824  DSN:  986-4824 
FAX:  (513)-476-4658 

Raymond  P  Covert 
Tecolote  Research  Inc. 

3601  Aviation  Blvd.,  #1600 
Manhattan  Beach  CA  90266 
OFF  TEL:  (310)-536-001 1 

Kenneth  E  Cox 
General  Dynamics 
3190  Fairview  Park  Drive,  #1 
Falls  Church  VA  22042-4523 
OFF  TEL:  (703)-284-9230 
FAX:  (703)-284-9244 


Lianna  Cruz 
JWAC 

18385  Frontage  Road 
Code  JN61 

Dahlgren  VA  22448-5500 

OFF  TEL:  (703)-663-1892  DSN:  249-1892 

Dianne  M  Cutshaw 
MARCORSYSCOM 
Code  PSA-0 

2033  Barnett  Ave,  Suite  315 

Quantico  VA  22134-5010 

OFF  TEL:  (703)-640-4455  DSN:  278-4455 

FAX:  (703)-640-2168 

L.  Patrick  Cyrus 
USAF  ASC/XRPC 
Bldg  450 

Wright-Patterson  AFB  OH  45433 

OFF  TEL:  (513)-255-5288  DSN:  785-5288 

FAX:  (513)-476-7155 

Karl  R  Dahlen 

Military  Professional  Resources  Inc. 

Hotel  Chamberlin,  Suite  701 
PO  Box  12 

Fort  Monroe  VA  23651 
OFF  TEL:  (804)-723-8053 
FAX:  (804)-723-4089 

Robert  A  Daigle 
US  Army  TACOM 
Fleet  Planning  Office 
AMSTA-TR-D  (MS  162) 

Warren  MI  48397-5000 

OFF  TEL:  (313)-574-6703  DSN:  786-6703 

FAX:  (810)-574-5201 

E-mail :  daigleb@tacom-emh  1 65  .army.mil 

Tung  M  Dang 
US  Army  PEO-GPALS 
P.O.  Box  1500 
Attn:  SFAE-GPL-GBR-P 
Huntsville  AL  35807 

OFF  TEL:  (205)-955-5992  DSN:  645-5992 
FAX:(205)-955-1867 

DR  Dale  M  Dannhaus 
US  Army  TRAC-WSMR 
Attn:  ATRC-WD 

White  Sands  Missile  Range  NM  88002 
OFF  TEL:  (505)-678-4617  DSN:  258-4617 
FAX:  (505)-678-5104 
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Elizabeth  A  Davies 

Army  Logistics  Management  College 

ALMC 

Fort  Lee  VA  23801 

OFF  TEL:  (804)-765-4250  DSN:  539-4250 
FAX:  (804)-765-4648 

Frank  Doiron 

Maryann  P  Dominiak 

HQ  AMC 

Attn:  AMCRM-E 

5001  Eisenhower  Ave 

Alexandria  VA  22333-0001 

OFF  TEL:  (703)-274-9082  DSN:  284-9082 

FAX:  (703)-274-8425 

Capt  Stuart  L  Domfeld 

ESC/TGN 

1 1  Eglin  Street 

Hanscom  AFB  MA  0 1 73 1  -2 1 20 

OFF  TEL:  (617)-377-8928  DSN:  478-8928 

FAX:  (6 17)-3 77-7447 

Eugene  R  Douglas 
Martin  Marietta  Air  Traffic  Systems 
475  School  Street,  SW 
MS-V62 

Washington  DC  20024 
OFF  TEL:  (202)-646-5813 
FAX:  (202)-646-2255 

Dennis  L  Doyle 
PRC,  Inc 

2750  Killamey  Drive 
Suite  200 

Woodbridge  V A  22192 
OFF  TEL:(703)-730-1951 
FAX:  (703)-730-1960 

Hubert  W  Drake 
ASl  Systems  International 
825  N  Downs,  Suite  C 
China  Lake  CA  93555 
OFF  TEL:  (619)-375-1442 
FAX:  (619)-375-0230 

Katherine  F  Drew 

Naval  Surface  Warfare  Center 

Code  U31 

Silver  Spring  MD  20903-5000 

OFF  TEL:  (202)-394-1457  DSN:  290-1457 

FAX:  (202)-394-1164 


Kristina  J  Ennis 
Carderock  Divsion/NSWC 
Code  1210 

Bethesda  MD  20084-5000 
OFF  TEL:  (3  01  >227-4699 
FAX:  (301  >227- 103 8 
E-mail:  ennis@oasys.dt.navy.mil 

DR  Henry  L  Eskew 

Center  for  Naval  Analyses 

440 1  Ford  Ave 

Alexandria  VA  22302 

OFF  TEL:  (703)-824-2254  DSN:  289-2638 

FAX:  (703)-824-2949 

CDR  Melvin  R  Etheridge  (Ret) 

Logistics  Management  Institute 
6400  Goldsboro  Rd 
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